AMENDMENTS TO THE CLAIMS 



Claims 1-8 (Canceled) 

9. (New) An ingredient cooking-operation recognition system comprising: 

a sensing means for acquiring observation data which includes at least either of an 
image around the hands of a person who performs a cooking operation and an 
environment sound that is produced by the cooking operation; 

a feature-quantity template in which various cooking operations that are 
predetermined for various ingredients are stored in a table form, together with a template 
certainty factor that is a certainty factor which is predetermined for each cooking 
operation of the ingredients; 

a cooking-flow creating means for, based on the observation data which is 
acquired by the sensing means, calculating an observation certainty factor which 
indicates the certainty factor of at least either of an ingredient that is cooked by the 
person and the cooking operation of the ingredient, and based on this observation 
certainty factor, creating a cooking flow for a dish which is prepared by the person; 

a primary certainty-factor calculating means for, based on the template certainty 
factor and the observation certainty factor, calculating a primary certainty factor which is 
a certainty factor for each of objects that make up the cooking flow which is created by 
the cooking-flow creating means; and 

a cooking-operation recognizing means for calculating a final certainty factor 
based on the primary certainty factor which is calculated by the primary certainty-factor 
calculating means, updating the feature-quantity template by considering this final 
certainty factor as a new template certainty factor, and based on this feature-quantity 
template, recognizing the ingredient and the cooking operation that correspond to the 
observation data which is acquired by the sensing means. 

10. (New) The ingredient cooking-operation recognition system according to claim 9, 
further comprising: 

a cooking-flow database which stores cooking flows which are created in advance 
for various dishes; and 
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a relevance-factor calculating means for, using flow matching, calculating a 
relevance factor that indicates which of the cooking flows that are stored in the cooking- 
flow database is closest to the cooking flow which is created by the cooking-flow 
creating means, 

the cooking-operation recognizing means recognizing an ingredient and a cooking 
operation, based on the relevance factor which is calculated by the relevance-factor 
calculating means and the primary certainty factor. 

1 1 . (New) The ingredient cooking-operation recognition system according to claim 9, 
further comprising: 

a table storing means for storing at least any one of: 

an ingredient table which is formed by an ingredient 
record that has a field where data that indicates an ingredient is 
stored and a field where various kinds of data on the ingredient are 
stored together with a template certainty factor that is 
predetermined for the data; 

a cooked-food table which is formed by a cooked-food 
record that has a field where data that indicates a cooking operation 
is stored and a field where various kinds of data on a change in an 
ingredient by the cooking operation are stored together with a 
template certainty factor that is predetermined for the data; and 

a cooking-operation table which is formed by a cooking- 
operation record that has a field where data that indicates a cooking 
operation is stored and a field where various kinds of data on an 
environment sound that is produced by the cooking operation are 
stored together with a template certainty factor that is 
predetermined for the data; 
a cooking-recipe database which stores data that indicates an ingredient which is 
used for various cooking recipes and data that indicates a cooking operation for the 
ingredient; 
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an extracting means for extracting all kinds of ingredients and all kinds of 
cooking operations which are stored in the cooking-recipe database; and 

a feature-quantity template creating means for creating, as a feature-quantity 
template, a table which includes, as items, each ingredient that is extracted by the 
extracting means and each cooking operation that is extracted by the extracting means, 

the feature-quantity template creating means writing, in an optional field of the 
fields which make up the feature-quantity template, as the template certainty factor, the 
greatest certainty factor of the template certainty factors which are stored in each field of 
at least any record of the ingredient record of the ingredient table that stores the data 
which indicates the ingredient that corresponds to the optional field, the cooked-food 
record of the cooked-food table that stores the data which indicates the cooking operation 
that corresponds to the optional field and the cooking-operation record of the cooking- 
operation table that stores the data which indicates the cooking operation that corresponds 
to the optional field. 

12. (New) The ingredient cooking-operation recognition system according to claim 9, 
characterized in that: 

the sensing means has a thermal camera and an optical camera; and 

an image which is acquired by the sensing means includes a thermal image which 

is photographed by the thermal camera and an optical image which is photographed by 

the optical camera. 

13. (New) The ingredient cooking-operation recognition system according to claim 
12, characterized in that the cooking-flow creating means: removes a background region 
from the optical image by executing a background difference processing on the optical 
image; removes, from the thermal image, as a background region, a region where the 
temperature is equal to, or higher than, a predetermined value; calculates an ingredient 
region which indicates an ingredient by multiplying the thermal image whose background 
region is removed and the optical image whose background region is removed; and 
calculates an observation certainty factor based on this ingredient region. 
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14. (New) The ingredient cooking-operation recognition system according to claim 9, 
characterized in that the cooking- flow creating means obtains a histogram of hue and a 
histogram of saturation from an image which is acquired by the sensing means, and 
estimates an ingredient by obtaining a correlation between these histograms and a color- 
feature template which is predetermined for each ingredient. 

15. (New) The ingredient cooking-operation recognition system according to claim 
10, characterized in that: 

the relevance-factor calculating means estimates a cooking recipe based on a 
relevance factor; and 

a guidance means is further provided for giving, to the person, guidance on the 
cooking recipe which is estimated by the relevance-factor calculating means. 

16. (New) A computer-readable recording medium which is recorded with an 
ingredient cooking-operation recognition program, characterized by allowing a computer 
to function as: 

a sensing means for acquiring observation data which includes at least either of an 
image around the hands of a person who performs a cooking operation and an 
environment sound that is produced by the cooking operation; 

a feature-quantity template in which various cooking operations that are 
predetermined for various ingredients are stored in a table form, together with a template 
certainty factor that is a certainty factor which is predetermined for each cooking 
operation of the ingredients; 

a cooking-flow creating means for, based on the observation data which is 
acquired by the sensing means, calculating an observation certainty factor which 
indicates the certainty factor of at least either of an ingredient that is cooked by the 
person and the cooking operation of the ingredient, and based on this observation 
certainty factor, creating a cooking flow for a dish which is prepared by the person; 

a primary certainty-factor calculating means for, based on the template certainty 
factor and the observation certainty factor, calculating a primary certainty factor which is 
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a certainty factor for each of objects that make up the cooking flow which is created by 
the cooking-flow creating means; and 

a cooking-operation recognizing means for calculating a final certainty factor 
based on the primary certainty factor which is calculated by the primary certainty-factor 
calculating means, updating the feature-quantity template by considering this final 
certainty factor as a new template certainty factor, and based on this feature-quantity 
template, recognizing the ingredient and the cooking operation that correspond to the 
observation data which is acquired by the sensing means. 
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